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Abstract - The Hierarchical Segmentation (HSEG) algorithm 
is an approach for producing high quality, hierarchically related 
image segmentations. The VisiMine image information mining 
system utilizes clustering and segmentation algorithms for 
reducing visual information in multispectral images to a 
manageable size. The project discussed herein seeks to enhance 
the VisiMine system through incorporating hierarchical 
segmentations from HSEG into the VisiMine system. 

I. Introduction 

The Hierarchical Segmentation (HSEG) algorithm is an 
approach for producing high quality, hierarchically related 
image segmentations. The HSEG algorithm is a hybrid of 
region growing and spectral clustering that produces a 
hierarchical set of image segmentations. 

Different image analysis applications often require 
different levels of image segmentation detail. Further, a 
particular application may require different levels of image 
segmentation detail depending on the image objects being 
segmented. Thus, a choice of several levels of image 
segmentation detail is often very useful. The HSEG 
algorithm automatically produces a hierarchical set of 
segmentations, based on detected convergence points. An 
analyst can select between segmentations at different levels in 
the segmentation hierarchy to produce the most appropriate 
segmentation for a particular application. 

The VisiMine image information mining system utilizes 
clustering and segmentation algorithms for reducing visual 
information in multispectral images to a manageable size. 
The indexing strategy extracts three levels of features: 1) 

pixel level, 2) polygon level, and 3) tile level features. The 
pixel level features are used for the extraction of higher-level 
features. Fast segmentation algorithms based on PDE 
methods are used to decompose image tiles into polygons. 

In the paper we compare the results of similarity searches 
utilizing features extracted using an enhanced version of 
CLARA algorithm (Clustering LARge Applications) in 
combination with statistical sampling with the results that use 
hierarchical image segmentation (HSEG) to derive class 


This project is supported by NASA's Intelligent Systems 
Program, NASA Headquarters Code R. 


and tile similarity searches based on spectral and textural 
properties. The use of HSEG features improves the precision 
and recall of similarity searches. The interactive label 
training and retrieval based on Bayesian models also benefits 
from hierarchical image segmentation approach. 

II. Hierarchical Segmentation 

We provide here a high level description of the HSEG 
algorithm. This description is a refinement of the description 
given earlier in [1] and follows that given in [2]: 

HSEG Basic Algorithm Description : 

1. Give each image pixel a region label and set the global 
criterion value, critval, equal to zero. If a pre- 
segmentation is provided, label each image pixel 
according to the pre-segmentation. Otherwise, label each 
image pixel as a separate region. 

2. Calculate the dissimilarity criterion value between each 
spatially adjacent region. 

3. Find the smallest dissimilarity criterion value and merge 
all pairs of spatially adjacent regions with this criterion 
value. 

4. Calculate the dissimilarity criterion value between all 
pairs of non-spatially adjacent regions (optional). 
Multiply these dissimilarity criterion values by 
spclust_wght (a preset weighting factor with default 
value 1.0). 

5. Merge all pairs of non-spatially adjacent regions with 
dissimilarity criterion value less than or equal to the 
criterion value found in step 3 (optional). 

6. If the number of regions remaining is less than the preset 
value chkregionSy go to step 7. Otherwise, go to step 2. 

7. Let preveritval — critval. Calculate the current global 
criterion value and set critval equal to this value. If 
preveritval = zero, go to step 2. Otherwise calculate 
cvratio = critval/prevcritval. If cvratio is greater than 
the preset threshold convfact, save the region label map 
from the previous iteration as a “raw” segmentation 
result. Also, store the region number of pixels list, 
region mean vector list and region criterion value list for 
this previous iteration. (Note: The region criterion value 
is the portion of the global criterion value contributed by 



the image pixels covered by the region.) If the number 
of regions remaining is two or less, save the region label 
map from the current iteration as the coarsest instance of 
the final hierarchical segmentation result and stop. 
Otherwise, go to step 2. 

The results stored in step 7 can be used in post-processing 
analysis to select a single segmentation from the 
segmentation hierarchy using an analysis tool such as the 
“Region Labeling Tool” described elsewhere [3]. In this 
paper we explore incorporating the hierarchical 
segmentations into the VisiMine image information mining 
system. 

A practical implementation of the HSEG algorithm 
requires either recursion or a coarse-to-fine spatial resolution 
processing scheme. Large-scale parallel processing is 
required to obtain reasonable processing times for either 
implementation scheme. Further, a “pixel switching” scheme 
must be employed in the recursive implementation in order 
avoid processing window artifacts, and a “pixel splitting” 
scheme must be employed in the coarse-to-fine approach 
when moving to finer spatial resolutions. A future paper will 
address these implementation issues. 

m. VisiMine Image Information Mining 

VisiMine project aims to provide the infrastructure and 
methodology required for the analysis of satellite images. In 
order to facilitate the analysis of large amounts of image data, 
we extract features of the images. Large images are 
partitioned into a number of smaller, more manageable image 
tiles. In addition to providing faster extraction of segments, 
partitioning allows fetching of just the relevant tiles when 
retrieval of only part of the image is requested. Then those 
individual image tiles are processed to extract the feature 
vectors. The VisiMine architecture supports three levels of 
features: 

1 . pixel level features, 

2. region level features, and 

3. tile level features. 

The feature extraction process starts with the analysis of 
spectral and textural properties at the pixel level. The 
numerical pixel data can be clustered in order to find a small 
number of classes. At the same time, the tile level features 
may be extracted, thereby creating histograms of the pixel 
classes for each tile. 

The extraction of region level features starts with a 
segmentation based on the algorithm presented in [4]. This 
function segments an input image into non-overlapping 
regions by minimizing an energy functional that trades off the 
similarity of regions against the length of their shared 
boundary. The geometrical properties of regions, such as 
image moments, are extracted. Based of the pixel features, 
the system creates histograms, mean, and standard deviation 
features for each region. 


Additional features are extracted using raster information, 
such as digital elevation maps. These features can be created 
at all three levels. 

A. Feature clustering using CLARA 

In order to build features that are used in the information 
retrieval and data mining functions, we perform unsupervised 
clustering using the CLARA algorithm [5]. The user chooses 
the number of classes, and each pixel is assigned to a single 
class. Textural information may be processed in the same 
way to extract textural classes. For tile and region level 
features, histograms of pixel distributions between classes are 
created. 

B. Texture Feature Extraction 

We extract pixel level texture features by using Gabor 
wavelets. In our comparison study of texture based 
classification, the Gabor features were judged to provide 
superior performance when compared to other texture 
analysis methods, such as edge attribute processing methods, 
circular simultaneous autoregressive model methods, or 
hidden Markov model methods [6]. In VisiMine, for each 

pixel we extract eight texture features CC i | J=0 7 by using 

Gabor Filters with kernels rotated by . To obtain rotation 
invariant features, we find the values of the autocorrelation 
function [6): 
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To minimize the range of the pixel index, we have chosen 
to compute values of the autocorrelation for n~0 r 4. These 
values correspond to the 0° and 90° differences in the 
orientation of the Gabor kernels. This shift should allow 
detection of urban road networks, which usually are 
correlated within a 90° rotation of the wavelet kernels. The 
texture features are extracted using two different scales. 
Rotation invariance can be observed, and urban regions are 
judged to be similar, regardless of the orientation of street 
networks. For example, the suburban area of New 
Westminster, in the Greater Vancouver area, is judged to be 
similar to East Vancouver, despite the fact that the principal 
directions of the street networks differ by about 30° for these 
two regions. The extraction of other microfeatures, such as 
frequency and orientation, also is possible [7], and we plan to 
perform more experiments with these features in the future. 

In addition to the extraction of Gabor texture features, the 
system provides for extraction of Haralick’s coocurrence 
features and Laws texture features. But, in our information 
retrieval experiments, we found the Gabor textures to be 
superior in terms of recall and precision levels. 



C. Similarity Search 

VisiMine uses an SQL-like query language that enables 
specification of the data mining task, the features that are to 
be used in the mining process, and any additional constraints. 
The system is capable of performing similarity searches 
based on any combination of features. A user can look for 
the most similar image tiles, or for the most similar regions 
based on a pattern tile or a pattern region. VisiMine allows 
arbitrary weighting of the features. The values of the features 
can be adjusted to have the range [0, 1], they can be 
multiplied by a specific value, or they can remain the same. 

IV. Results 

In our experimentation with region based searches we 
looked only for regions having areas larger than 3000 pixels. 
The feature values were scaled to the range [0,1]. For 
CLARA algorithm we used 25 clusters, which warranties 
good results for this algorithm. The number of clusters for the 
RHSEG algorithm varied based on the level. For the 
experiments we used only histograms based on clusters that 
contained more than 1000 pixels from 4096 x 4096 Landsat 
TM image from over the Washington, DC/Baltimore, MD 
area. We compared the results of similarity searches based 
on CLARA features with searches based on HSEG features, 
or on a combination of the these features with Gabor texture 
features. 

In the oral presentation we will present detailed results 
using tables and graphs. Unfortunately, space requirements 
do not allow inclusion of a complete set of detailed results in 
the proceedings paper. An example of such results is 
provided here in Table 1. 

The best results are obtained using level 1 and 2 of HSEG 
and adding texture features improves the results of the 
majority of the queries. The best results depend on the type 
of the query. We believe that using the relevance feedback 
for the automated selection of the subsets of the features 
should improve the accuracy of search. In the future we are 
planning to use the relevance feedback for automated 
selection of the subsets of the features should improve the 
accuracy of search. 

V. Future Work 

In the future, we are planning to use the relevance 
feedback for automated selection of the subsets of the 
features should improve the accuracy of search. We will also 
investigate methods for automatically and adaptively 
selecting the hierarchical segmentation levels utilized. 

The results of information retrieval can be improved using 
the Synthetic Aperture Radar (SAR) data. We are planning 
to perform feature extraction and classification/mining 
examples based on SRTM data for the Baltimore area. We 
also want to compare the topology of the HSEG intermediate 


outputs for scenes at two different times for the same area. 

The preliminary analysis shows the patterns that reinforce the 

classification process and help identify outliers in the data. 
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Table I 

Similarity Search Results using various Features 


Feature 

Average Precision 

25 spectral CLARA clusters 

0.7522999 

25 spectral CLARA clusters 
with 10 Gabor texture clusters 

0.7304257 

HSEG level 1 region histograms 

0.7746878 

HSEG level 2 region histograms 

0.7728281 

HSEG level 2 clusters 
with 10 Gabor texture clusters 

0.7649297 

HSEG level 3 region histograms 

0.7207362 

HSEG level 3 clusters 
with 10 Gabor texture clusters 

0.7528406 

HSEG level 4 region histograms 

0.7061142 

HSEG level 6 region histograms 

0.6435331 

HSEG level 7 region histograms 

0.4792615 




